DMftNo. 43701-00034 



United States Patent Application 

For 

Security Apparatus and Method 

Assignee: Kabusbiki Kaisha Toshiba, Japan 
Inventors: Kaoru Suzuki; Kazuhiro Fukui; Hisashi Kazama; Osamu Yamaguchi; 
Eiji Tanaka; Yasuhiro Taniguchi, all of Tokyo, Japan 



Squire, Sanders & Dempsey L.L.P. 
801 South Figueroa Street, 14 th Floor 
Los Angles, California 90017-5554 
Tel.: (2 13) 624-2500 
Fax: (213)623-4581 

Attorney Matter No. 43701.00034 



SECURITY APPARATUS AND METHOD 

Background of the Invention 

[0001] 1. Field of the Invention: 

The present invention relates to a security apparatus and a method for supplying a 
service to a user by confirming a use situation of the user and infringement situation of 
non-user in order to keep safety and reliability for the service. 
[0002] 2. Description of the Related Art: 

A method to specify the user in service-use area by electrical means and to 
execute security for each kind of service is very important in present electrical society. 
Various kinds of security means, for example, memorized number, magnetic card, IC 
card, wireless card, sound comparison, fingerprint comparison, retina comparison and 
face comparison, have been used. However, the memorized number, the magnetic card, 
the IC card and the wireless card specifies the user by information only the user knows or 
with a memory device key for only the user. Therefore, they are in danger of being used 
unjustly. 

[0003] On the other hand, sound comparison, fingerprint comparison, retina 
comparison, and face comparison are effective personal confirmation means to detect 
[defect] an other's unjust use because the user's physical features are used as a key. 
However, the personal confirmation is only used at the begining of service. After the 
service begins, the service can not be interrupted. Therefore [Therfore], even if an other 
person changes for the user after the service is begun, the service can not be interrupted. 
In short, this kind of service includes danger that the other person utilizes the service 
unjustly [unjusty]. This kind of problem is included in the memorized number, the 
magnetic card, the IC card and the wireless card. 

[0004] As mentioned [mentiored] above, the known security apparatuses include 
the following problems. 
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[0005] (1) After security is cancelled by confirming the user, security function is not 
executed while supplying the service. Therefore, even if an other person changes for the 
user, the service is still supplied continuously. 

[0006] (2) While supplying the service to the user, it often happens that an other 
person watches information not to be disclosed on display from behind the user. This 
problem occurs because security is not provided in the physical space surrounding the 
user and the apparatus even if security is executed inside the apparatus electrically. 
Especially, this problem happens when the service is supplied to the user in public place 
in which other persons are not excluded. 

Invention Summary 

[0007] It is an object of the present invention to provide a security apparatus and 
method to supply a service to the user while safety and reliability for the service are kept. 
[0008] According to the present invention, there is provided a security apparatus for a 
device supplying a service to a user, comprising: person discrimination means for 
recognizing a user requesting the service; use situation decision means for deciding 
whether the user is under a situation to use the service; infringement situation decision 
means for detecting whether a non-user intrudes into a use area the service to decide 
whether of the service is infringed; service control means for supplying the service to the 
user in case said person discrimination means recognizes the user, and for controlling a 
supply of the service if said use situation decision means decides the user is not under the 
situation to use the service or if said infringement situation decision means decides that 
the security of the service is infringed. 

[0009] Further in accordance with the present invention, there is provided a security 
method associated with supplying a service to a user, comprising the step of: recognizing 
a user desiring the service: supplying the service to the user if the user is recognized; 
deciding whether the user is under a situation to use the service; deciding whether a non- 
user intrudes into a use area of the service; and controlling the supply of the service if the 
user is not under the situation to use the service or if the non-user is intruding into the use 
area of the service. 

[0010] Further in accordance with the present invention, there is also provided a 
computer readable memory containing computer-readable instructions to supply a service 
to a user, comprising: instruction means for causing a computer to recognize a user 
desiring the service; instruction means for causing a computer to supply the service to the 
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user if the user is recognized; instruction means for causing a computer to decide whether 
the user is under a situation to use the service; instruction means for causing a computer 
to decide whether a non-user intrudes into a use area of the service; and instruction means 
for causing a computer to control a supply of the service if the user is decided not under 
the situation to use the service or the non-user is decided to intrude into the use area of 
the service. 

Brief Description of the Drawings 

[0011] FIG. 1 is a block diagram of the security apparatus according to the present 
invention. 

[0012] FIG. 2 is a block diagram of the situation detection section of the security 
apparatus according to the present invention. 

[0013] FIG. 3 is a flow chart of processing of a service supply permission section 
according to a first embodiment of the present invention. 

[0014] FIG. 4 is a flow chart of processing of the service supply permission section 
according to a second embodiment of the present invention. 

[0015] FIG. 5 is a flow chart of processing of the service supply permission section 
according to a third embodiment of the present invention. 

[0016] FIG. 6 is a flow chart of processing of the service supply permission section 
according to a fourth embodiment of the present invention. 

[0017] FIG. 7 is a block diagram of the security [sequrity] apparatus according to a 
fifth embodiment of the present invention. 

[0018] FIG. 8 is a flow chart of processing of the service supply permission section 
according to the fifth embodiment of the present invention. 

[0019] FIG. 9 is a flow chart of processing of the service supply permission section 
according to a sixth embodiment of the pres sent invention. 

[0020] FIG. 10 is a flow chart of processing of the service supply permission section 
according to a seventh embodiment of the present invention. 

[0021] FIG. 1 1 is a block diagram of the security [sequrity] apparatus according to an 
[a] eighth embodiment of the present invention. 

[0022] FIG. 12 is a flow chart of processing of the service supply permission section 
according to the eighth embodiment of the present invention. 

[0023] FIG. 13 is a schematic diagram of detection of visual lines of the user in the 
image according to the eighth embodiment of the present invention. 
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[0024] FIG. 14 is a flow chart of processing of the service supply permission section 
according to a ninth embodiment of the present invention. 

[0025] FIG. 15 is a schematic diagram of detection of visual lines of the non-user in 
the image according to the ninth embodiment of the present invention. 
[0026] FIG. 16 is a flow chart of processing of the service supply permission section 
according to a tenth embodiment of the present invention. 

[0027] FIG. 17 is a schematic diagram of supply method for the security apparatus by 
software according to the present invention. 

Detailed Description of the Preferred Embodiments 

[0028] The embodiments of the present invention are described below with reference 
to the drawings. FIG. 1 is a block diagram of the security apparatus according to the 
present invention. The security apparatus of the present invention is comprised of a 
situation detection section 1, a service supply permission section 2, and a service supply 
section 3. The situation detection section 1 discriminates the user in service-use area, 
detects whether the user is under a situation to use the service (use situation) and detects 
whether a non-user infringes the security of the user (infringement situation). The 
detection result of the use situation and the infringement situation is outputted to the 
service supply permission section 2 as person information. In this place, the situation that 
the user can utilize the service represents that the user is confirmed to be permitted the 
supply of the service and the user continuously utilizes the service. A situation that the 
user continuously utilizes the service represents a status that the user actually interacts 
with the service application (the user inputs operation to system or the user is supplied 
output from the system). A status of non-interaction within predetermined time indicates 
thinking chattering, looking away. Another status indicates when the user moves away 
from service use-space within a predetermined time. 

[0029] The situation that a non-user infringes the security of the user represents that a 
person other than the confirmed user intentionally or accidentally uses the service while 
the service is supplied to the confirmed user. The other person can then interact with the 
operation or information to be disclosed only to the confirmed user. When the user 
interacts with the service application of the system through a display, the other person 
looks on the display behind the user. Otherwise, the other person intends to interact with 
the service application in the user's absence within the predetermined time. 
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[0030] In order to detect each situation, the situation detection section 1 monitors 
service-use area using image or sound. Especially, the use situation of the user and the 
infringement situation of the security are detected by the image (person 
detection/direction of face detection/person confirmation) and the sound (person 
detection/person specification/ chattering detection). Monitoring may be wireless. 
[0031] The service supply permission section 2 controls start/interruption/interruption 
and cancellation/end of the service for the service supply section 3 according to the 
detection result of the situation detection section 1. In this case, the service supply 
permission section 2 starts the supply of the service when the user is specified in the 
service-use area. The service supply permission section 2 controls the supply of the 
service in at least one of the cases where the user does not utilize the service or where the 
security of the user is infringed. Especially when the user does not utilize the service 
during the supply of the service, the supply of the service finishes. If the security of the 
user is infringed by another person the supply of the service interrupts [interupts] until the 
infringement is relieved. The service supply section 3 supplies various kinds of service 
under the control of the service supply permission section 2. For example, an 
uninhabited ATM ( automatic [antomatic] teller machine), a portable terminal, or a 
computer used by several persons may comprise the service supply section 3. 
[0032] FIG. 2 is a block diagram of the situation detection section 1 . In this case, an 
image is used as a way to discriminate the user, detecting the use situation, or detect an 
infringement situation. The situation detection section 1 is comprised of an image input 
section 11, a person area detection section 12, a person comparison section 13, a use 
situation decision section 14 and an infringement situation decision section 15. The 
image input [imput] section 1 1 inputs an image through a TV camera. The TV camera is 
set in the service use area for continuously observing the user to be supplied the service. 
The person area detection section 12 analyzes images input by the image input section 1 1 
and monitors the secure area from the input image. For example, a difference between 
the input image and background image (uninhabited area) indicates a person present in 
the secure area. The person area detection section 12 outputs person detection 
information Dl when the person present is detected and outputs non-detection 
information Dl when the person present is not detected. 

[0033] The person comparison section 13 compares the person present area image 
with the content of a person comparison dictionary to specify which person is present in 
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the secure area image. The person comparison dictionary previously stores at least 
comparison information of persons permitted to use the service. The person comparison 
section 13 outputs person discrimination information D2 representing whether the person 
present coincides with a person permitted to use the service. If the person present 
coincides with a person permitted to use the service in the person comparison dictionary, 
the person present is registered as the user who can be supplied the service. 
[0034] The use situation decision section 14 detects whether the user is under a 
situation capable of using the service. After the user of the person area is registered and 
if the registered user is now included in the person discrimination information D2, the 
user is decided to be capable of using the service. The use situation decision [decison] 
section 14 outputs use situation information D3 representing whether the user is capable 
of using the service. 

[0035] The infringement situation decision section 15 detects whether a non-user 
infringes the security of the service. If a person other than the user utilizing the service 
now is included in the person discrimination information D2, the infringement situation 
decision section 15 decides that the security of the service is infringed, and outputs 
infringement situation information D4 indicates a breach of security. The person 
detection/non-detection information Dl, the person discrimination information D2, the 
use situation information D3, the infringement situation information D4 are called person 
information. The person information is read out by the service supply permission section 
2 (step SI in FIG. 3 explained afterward) and referred at steps S3, S4, S8, Sll in FIG. 3. 
[0036] FIG. 3 is a flow chart of processing of the service supply permission section 2. 
As mentioned above, the situation detection section 1 discriminates the user in the 
service-use area and monitors the use situation and the infringement situation using the 
image inputted from the TV camera. The person information as detection result of the 
situation detection section 1 is outputted to the service supply permission section 2. By 
using the person information, the service supply permission section 2 executes a security 
controlling process to the service supply section 3. First, the service supply permission 
section 2 obtains the person information from the situation detection section 1 (step SI). 
When the supply of the service begins ("No" at step S2), the service supply permission 
section 2 monitors the security area in the service use area according to the person 
detection/non-detection information Dl (step S3). If a person area is not detected ("No" 
at step S3), processing of the service supply permission section 2 is returned to step SI. 
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On the other hand, if a person is detected ("Yes" at step S3), the service supply 
permission [premission] section 2 decides whether the person is permitted as a user (step 
S4). As a result, in case the person is not permitted as a user ("No" at step S4), the 
service supply permission section 2 rejects the supply of the service (step S5) and returns 
processing to step SI. In case the person is permitted as a user ("Yes" at step S4), the 
service supply permission section 2 registers the person as the user (step S6). After that, 
the use situation of the person is continuously monitored. The service supply permission 
section 2 begins the supply of the service (step S7) and returns the processing to SI. 
[0037] The above process steps (step S1-S7) are the same as normal confirmation of 
security (from confirming the user to beginning [begining] the supply of the service). In 
this place, after the supply of the service is begun, the registered user exists in the 
service-use area and the registered person must be detected at step SI. After the supply 
of the service is begun the processing is returned to step S8 through steps SI and S2. 
[0038] The service supply permission section 2 decides whether the registered person 
is using the service according to the use situation information D3 (step S8). As a result, 
if the situation of using the service is detected, for example, a user's absence continues 
above a predetermined time ("No" at step S8), the service supply permission section 2 
terminates the supply of the service immediately (step S9). Next, the registration of the 
user is cancelled (step S10) and the processing returns to step SI in order to register a 
new user. 

[0039] On the other hand, the service supply permission section 2 decides whether a 
non-user infringes the security of the user according to the infringement situation 
information D4 (step Sll). As a result, if the situation that a non-user infringes the 
security is detected, for example, another person watches the display behind the user 
during the supply of the service ("Yes" at step Sll), the service supply permission 
section 2 temporarily interrupts the service to maintain security until the security is not 
infringed (step SI 2). If a situation that the security is not infringed is detected during the 
interruption of the service ("Yes" at step SI 3), the service supply permission section 2 
cancels the interruption of the service and begins the supply of the service (step SI 4). 
[0040] In this way, in case the user permitted the service is specified in the service 
use area, the supply of the service is begun. During the supply of the service, if the user 
is not using the service or if the security is breached, the supply of the service finishes or 
interrupts. Accordingly, the security is maintained during all periods of beginning 
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[begining]/supplying/end of the service. For example, in case the registered user is 
confirmed and supplied money through an Automatic Teller Machine, or the user 
operates a handy terminal by watching secret information on a display, the security of the 
user is effectively protected according to the present invention. 
[0041] Next, additional embodiments of the present invention will be explained. 
[0042] (a) ease of monopoly of terminal/improvement of common use of terminal 
(second embodiment) 

[0043] In the first embodiment, another person can not use the terminal while the user 
is supplied the service through the terminal. Therefore, the other person must wait to use 
the same terminal until [untill] the first user finishes. In short, in the first embodiment, 
use by a specified user for the terminal excludes use by another [an other] person. 
[0044] In a second embodiment, in case the situation for the user not capable of using 
the service is detected before the supply of the service for the user finishes, the supply of 
the service is reserved (interrupted) and the new user can be supplied the service from the 
beginning [begining]. As a result, during the interruption of the service, a different 
service is supplied to another user. When the original user comes back to the terminal, 
the reserved service is supplied to the original user again. Therefore, while each service 
is not completed respectively, plural users can be supplied each service from the same 
terminal one after another. Additionally, if the original user does not come back to the 
same terminal, the reserved service is finished and memory area corresponding to the 
reserved service is relieved to save memory resources. 

[0045] FIG. 4 is a flow chart of processing of the service supply permission section 
according to the second embodiment. In this case, assume that a computer network is 
commonly used by plural users and a plurality of terminals connected to the network 
provide a security function. In FIG. 4. steps S15, SI 6, SI 7, S18 are newly added in 
comparison with FIG. 3. Each user is confirmed respectively and supplied the service 
through a respective terminal (step SI, S2). 

[0046] In this place, assume that a user A creates a document using a terminal A and 
leaves from the terminal A without completing the work. The terminal A detects that the 
user A is not using the service ("No" at step S8) and that the supply of the service is not 
completed ("No" at step SI 5). The service of creating the document at the present time is 
stored (reserved) in a memory area of the terminal and the supply of the terminal is 
interrupted (step SI 6, SI 7). In this case, the situation for the user not using the service is 
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detected according to the use situation information D3. Completion situation of the 
supply of the service is decided by completion notice sent by the user. Next, a user B is 
confirmed by the terminal [teminal] A and works through the terminal A. Assume that 
when the user A comes back to the terminal A, the user B goes away from the terminal A. 
The terminal A newly reserves the results of the service for the user B (steps S8, SI 5, 
SI 6, SI 7) and begins the supply of the reserved service to the user A (steps S8, SI 1, SI 3. 
514). The memory area to store the result of the service is commonly used by each 
terminal on the network. Therefore, if the user B is confirmed by a terminal B (steps S4, 
S6. S7), the reserved result of the service for the user B is supplied through the terminal 
B (steps S8, SI 1, SI 3, S14). On the other hand, if the user A does not come back to the 
terminal A during a predetermined time ("Yes" at step SI 8), the terminal A finishes the 
service of the user A immediately and relieves the memory area corresponding to the 
service result of the user A (steps S9, S10). In this way, a situation of storing many 
service results in memory area is prevented. The predetermined time as decision 
[decison] standard at step SI 8 is freely set by the kind of the service. 
[0047] As mentioned above, in the second embodiment, the supply of the service for 
each user is reserved until the work of each user is completed. In short, a right service- 
use generated by confirming the user is assured until [untill] the user's work is 
completed. The user B does not have to wait until the user A finishes using the terminal. 
By changing the user A for the user B, the user B can use the terminal without 
infringement for the security of the user A. 

[0048] (b) Extension [Extention] for unspecified user (third embodiment) 
[0049] In the first embodiment, in order to be supplied the service, the user must be 
confirmed as a registered person. In short, if the user is not previously registered, he can 
not be supplied the service. However, there is a service to be used by many and 
unspecified persons. Therefore, in a third embodiment, the service is supplied to an 
unspecified person without confirmation. At that time, a comparison dictionary of the 
unspecified person is generated and registered. During the supply of the service, the user 
(the unspecified person) is discriminated from other users by referring to the comparison 
dictionary. Therefore, the service is supplied to many and unspecified persons while the 
security is maintained. 

[0050] FIG. 5 is a flow chart of processing of the service supply permission section 2 
according to the third embodiment. In FIG. 5. step S19 is newly added instead of steps 
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S4, S5 in comparison with FIG. 3. When the supply of the service is begun ("No" at step 
S2), the service supply permission section 2 detects the person in the service use area 
according to the person detection/non-detection information Dl (step S3). If the person 
is detected ("Yes" at step S3), the service supply permission section 2 generates the 
person comparison dictionary (step SI 9) and registers the person as a new user (step S6). 
After that, the use situation of the new user is continuously monitored, and the service 
supply permission section 2 begins to supply the service (step S7). In this way, the 
service is supplied to an unspecified person without confirmation and the person 
comparison dictionary is generated to be registered. Therefore, a new user except for the 
pre-registered user is able to be supplied the service. In this case, during the supply of 
the service for the new user, he is discriminated from another person by referring the 
person comparison dictionary in order to maintain the security. 
[0051] (c) Warning of infringement (fourth embodiment) 

[0052] In the fourth embodiment, when an infringement situation of the security is 
detected, a warning is sent to the user. In this case, the service supply permission section 
2 controls [contrails] the beginning [begining] and end of the service and does not 
execute interruption and cancellation of the service in case of infringement of the 
security. 

[0053] FIG. 6 is a flow chart of processing of the service supply permission section 2 
according to the fourth embodiment. In FIG. 6, step S20 is a newly added step instead of 
step S12 in comparison with FIG. 3. First, when the situation detection section 1 detects 
the infringement situation of non-user ("Yes" at step SI 1), the service supply permission 
section 2 sends a warning of the infringement to the user (step S20). Concretely 
speaking, the warning is outputted through an image (warning sentence/mark, controlling 
of color/brightness/blink) or sound (speaking of warning sentence/warning buzzer). 
When the user receives the warning, it is possible for the user to protect his security by, 
for example, turning off a power supply of the terminal or hiding a display of the 
terminal. 

[0054] (d) Interactive controlling in case of warning (fifth embodiment) 
[0055] In the fifth embodiment, by using input means (key board) and output means 
(display), the supply of the service is controlled by the user's operation for the warning. 
FIG. 7 is a block diagram of the security apparatus according to the fifth embodiment. In 
FIG. 7. an interactive control section 4 is a new section in comparison with FIG. 1. FIG. 
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8 is a flow chart of processing of the service supply permission section 2 according to the 
fifth embodiment. In FIG. 8. steps S20, S21 are new added steps in comparison with 
FIG. 3. 

[0056] The interactive control section 4 receives an indication of the user for the 
warning of the infringement and controls [controlls] the supply of the service according 
to the indication. The kind of the indication is continuation/interruption/end of the 
service. The user selects the kind of the indication by, for example, using a keyboard. In 
FIG. 8, during the supply of the service, if the situation detection section 1 detects 
infringement by a non-user ("Yes" at step Sll), the service supply permission section 2 
sends the warning to the user (step S20). After sending the warning, the interactive 
control section 4 receives an indication of the user (step S21), the service supply 
permission section 2 controls [controlls] the supply of the service according to the 
indication. 

[0057] If the user indicates continuation of the service ("CONTINUATION" at step 
S21), the interactive control section 4 outputs the indication of continuation to the service 
supply permission section 2. The service supply section 3 continues to supply the service 
according to the indication of continuation. 

[0058] If the user indicates interruption of the service ("INTERRUPTION" at step 
S21), the interactive control section 4 outputs the indication of interruption to the service 
supply permission section 2. The service supply section 3 interrupts the supply of the 
service according to the indication of the interruption. 

[0059] If the user indicates end of the service ("END" at step S21), the interactive 
[interactive] control section 4 outputs the indication of end to the service supply 
permission section 2. The service supply section 3 finishes the supply of the service. 
[0060] In this way, when the user receives the warning of infringement, the user can 
select continuation/interruption/end of service by using interactive function. As a result, 
if the user works through equipment in an office, whenever an office colleague (person 
not necessary for security) stands behind the user, interruption of the work is excluded. 
(In short an interruption , [a change for the worse] of use[-feeling] of the equipment is 
avoided.) Otherwise, when using the equipment in a public place, interruption of the 
service is indicated by the user for the warning of a non-user standing behind the user. 
[0061] In the fifth embodiment, the user can select the indication to keep security in 
case of infringement of the security. After selecting the indication, the apparatus 
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(equipment) can automatically keep the security of the user. In this case, if the warning 
of the infringement is outputted as an image, the user understands circumstances around 
the user by watching the image (the user and non-user are included). The user can select 
the indication (continuation/interruption/end of the service) according to the 
circumstances. 

[0062] (e) Default action of neglecting the warning (sixth embodiment) 
[0063] In the sixth embodiment, in case the indication is not selected by the user 
within predetermined time after generating the warning, the apparatus automatically 
executes predetermined (default) controlling of the supply of the service. This 
predetermined controlling is freely set by the user. 

[0064] FIG. 9 is a flow chart of processing of the service supply permission section 2. 
In FIG. 9, steps S20, S21, S22, S23, S24 are added steps in comparison with FIG. 3. The 
construction of the security apparatus of the fifth embodiment is the same as in FIG. 7. 
In FIG. 9, during the supply of the service, if the situation detection section 1 detects an 
infringement situation ("Yes" at step Sll), the service supply permission section 2 sends 
the warning to the user (step S20). After the warning, the interactive control section 4 
receives indication of the user. In this case, if no indication of the user is received above 
predetermined time (step S22, S23), the interactive control section 4 outputs the purport 
of no indication to the service supply permission section 2. The service supply 
permission section 2 controls [controlls] the supply of the service according to a 
predetermined control (step S24). For example, if continuation of the service is set as the 
predetermined control ("CONTINUATION" at step S24), the service supply section 3 
continues to supply the service. If interruption of the service is set as the predetermined 
control ("INTERRUPTION" at step S24), the service supply section 3 interrupts the 
supply of the service. If end of the service is set as the predetermined controlling 
("END" at step S24). the service supply section 3 finishes the supply of the service. On 
the other hand, if the indication of the user is received within the predetermined time 
("Yes" at step S22), the interactive control section 4 receives the indication (step S21) 
and the service supply permission section 2 controls [controlls] the supply of the service 
according to the indication. In this case, if the user indicates continuation of the service 
("CONTINUATION" at step S21), the interactive control section 4 outputs the 
continuation indication to the service supply permission section 2. The service supply 
section 3 continues to supply the service. If the user indicates interruption of the service 
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("INTERRUPTION" at step S21), the interactive control section 4 outputs the 
interruption indication to the service supply permission section 2. The service supply 
section 3 interrupts the supply of the service. If the user indicates end of the service 
("END" at step S21), the interactive control section 4 outputs the end indication to the 
service supply permission section 2. The service supply section 3 finishes the supply of 
the service. 

[0065] In this way, if the user neglects a warning of infringement, the security of the 
user is kept according to a predetermined control (action of default). Otherwise, it is 
possible to control the supply of the service different form the predetermined control for 
some reasons by a user's indication in case of the warning. For example, in an office, as 
for the warning generated whenever a colleague stands behind the user operating the 
equipment, the user does not have to indicate continuation of the service. The 
predetermined control is already set as continuation of the service. In this case, the user 
can continue the operation while neglecting the warning. In case the user operates the 
equipment in a public place, the predetermined control is set as interruption of the 
service. In this case, security of the user for the equipment is protected while he neglects 
the warning. 

[0066] (f) Security degree by unit of service/information (seventh embodiment) 
[0067] In the above embodiments, predetermined security is activated for service or 
information. However, security degree is actually different by unit of 
service/information. If a fixed security degree is activated, useless interruption of the 
service is often executed. This happens in case of plural workers using information 
equipment. Accordingly, it is desired that the security is activated according to the kind 
of the service/information. For example, assume that electric mail is transmitted using a 
PC (personal computer) or a document is created using WP (word processor). Plural 
services are then supplied on the display of the equipment. The electric mail represents a 
private letter and its security degree is high. On the other hand, the security degree of the 
document by WP differs according to the kind of the document. While the user creates 
the document whose security degree is low, the security degree of the document grows by 
adding another important document. In this way, the security degree is dynamically 
changed according to the kind of the service/information. 

[0068] In the seventh embodiment, the security degree is changed by unit of the 
service/information. FIG. 10 is a flow chart of processing of the service supply 
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permission section 2 according to the seventh embodiment. In FIG. 10, step S25 added 
between SI 1 and S 12 is a different step in comparison with FIG. 3. During the supply of 
the service, if the situation detection section 2 detects infringement of the security ("Yes" 
at step SI 1), the service supply permission section 2 controls [controlls] the supply of the 
service according to the security degree set by the kind of service/information (step S25). 
In this case, it may be possible that the security degree is set to a mail tool to display the 
mail but not set to a WP tool. For example, when another person watches the display 
behind the user, the mail tool is only interrupted (the mall window is closed) and the WP 
window is continuously opened. 

[0069] As for setting the security degree, for example, the user sets disclosure- 
possible at the head portion of mail to be sent, A window showing the mail tool does not 
react to the infringement of the security. As for both sending side or receiving side, the 
mail is able to remain disclosing. 

[0070] If the user indicates a strict secret at the header of a document, the WP 
window to display the document reacts to the infringement of security. For example, the 
security [sequrity] degree for creating the document is dynamically changed by adding 
other important document. In this way, even if the service is supplied to the same object, 
the supply of the service is interrupted or continued according to the content (for 
example, regular document or strict secret document). 
[0071] (g) detection of visual line 1 (eighth embodiment) 

[0072] In the above embodiments, when another person watches the display behind 
the user creating a document, the infringement of security of the user is detected. 
However, it is not necessary to pay attention to the other person, but to pay attention to 
the visual line (or line of sight) of the user. For example, when the visual line of the user 
leaves the screen of the display, the window may be closed. In the eighth embodiment, 
by detecting a visual line of the user as use situation, the supply of the service is 
interrupted when the visual line leaves the screen. 

[0073] FIG. 1 1 is a block diagram of the situation detection section 1 according to the 
eighth embodiment. A visual line detection section 16 is an added section in comparison 
with FIG. 2. The visual line detection section 16 detects movement of the visual line of 
the user in case the user (registered) is included in the person discrimination information 
D2. For example, the face area of the user is extracted from the image and the eye area is 
extracted from the face area. The visual line is pursued according to the eye area. In this 
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case, if a direction of the visual line is toward a predetermined area (visual area of the 
screen), an indication is outputted. If the direction of the visual line is not toward the 
predetermined area, another indication is outputted. These indications represent visual 
line information D5. 

[0074] In this way, in the eighth embodiment, the visual line information D5 is 
outputted to the service supply permission section 2 as one of the person information. 
FIG. 12 is a flow chat of processing of the service supply permission section 2 according 
to the eighth embodiment. In FIG. 12, step S26 is added instead of step Sll in 
comparison with FIG. 3. During the supply of the service, assume that the situation 
detection section 1 detects that the user is under a situation to be supplied the service 
("Yes" at step S8). In this case, the visual line detection section 16 keeps watch for the 
movement of the visual line of the user. If the direction of the visual line departs from 
the predetermined area of the display ("Yes" at step S26), the situation detection section 

1 outputs the visual line information D5 to the service supply permission section 2. The 
service supply permission section 2 controls the service supply section 3 to interrupt the 
supply of the service (step SI 2). FIG. 13 is a schematic diagram of detection of the 
visual lines of the user in the image according to the eighth embodiment. As shown in 
FIG. 13, the visual lines of the user are detected from position of black eyes of the user's 
face in the image. 

[0075] In this way, in the eighth embodiment, detection of the visual line of the user 
controls [controlls] the supply of the service. Therefore, when another person walks 
behind the user, security of the service is kept only if the visual line of the user departs 
from the display. 

[0076] (h) Detection of visual line 2 (ninth embodiment) 

[0077] In the eighth embodiment, a direction of the visual line of the user controls 
[controlls] the supply of the service. However, if another person looks furtively at the 
display behind the user, the supply of the service can not be controlled. If the user turns 
his eyes away from the display whenever the other person walks behind the user, the 
supply of the service is interrupted to every time. Therefore, in the ninth embodiment, 
detection of the visual line of the non-user controls [controlls] the supply of the service. 
[0078] FIG. 14 is a flow chart of processing of the service supply permission section 

2 according to the ninth embodiment. In FIG. 14, step S27 is added step between step 
Sll and step S12 in comparison with FIG. 3. In this case, the visual line detection 
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section 16 detects the visual line of the non-user. In case the person is not discriminated 
by referring to the person discrimination information D2 (the person is decided to be a 
non-user), the movement of the visual line of the non-user is detected. In the same way 
as in the eighth embodiment, the face area is extracted from the image including the non- 
user and the eye area is extracted from the face area. The visual line of the eye area is 
pursued. If a direction of the visual line of the non-user points to a predetermined area 
(visible area of the display), an indication is outputted. If the direction of the visual line 
of the non-user does not point to the predetermined area, another indication is outputted. 
This purport is represented as the visual line information D5. During the supply of the 
service, assume that the situation detection section 1 detects that the non-user infringes 
the security of the non-user ("Yes" at step Sll). In this case, the visual line detection 
section 16 keeps watch for the move of the visual line of the user. If the direction of the 
visual line of the non-user points to the predetermined area, for example, the non-user 
watches the display behind the user ("No" at step S27), the situation detection section 1 
outputs the visual line information to the service supply permission section 2. The 
service supply permission section 2 controls [contrails] the service supply section 3 to 
interrupt the supply of the service. FIG. 15 is a schematic diagram of detection of the 
visual lines of the non-user in the image according to the ninth embodiment. As shown 
in FIG. 15, the visual lines of the non-user are detected from position of black eyes of the 
non-user's face in the image. 

[0079] In this way, in the ninth embodiment, the detection of the visual line of the 
non-user controls [contrails] the supply of the service. Therefore, in case the non-user 
looks furtively at the display behind the user, the security of the service for the user is 
kept. 

[0080] (i) Detection of visual line 3 (tenth embodiment) 

[0081] In the tenth embodiment, detection of the visual line of the user and the non- 
user controls [contrails] the supply of the service. FIG. 16 is a flow chart of processing 
of the service supply permission section 2 according to the tenth embodiment. In FIG. 
16, step S26 and step S27 are added steps in comparison with FIG. 3. During the supply 
of the service, assume that the situation detection section 1 detects that the user is under a 
situation to be supplied the service ("Yes" at step S8). In this case, the visual line 
detection section 16 detects movement of the visual line of the user. If the visual line of 
the user departs from the predetermined area, for example, the user turns his eyes away 
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from the display ("Yes" at step S26), the situation detection section 1 outputs the visual 
line information D5 to the service supply permission section 2. Then, the service supply 
permission section 2 controls [controlls] the service supply section 3 to interrupt the 
supply of the service (step SI 2). 

[0082] On the other hand, assume that the direction of the user points to the 
predetermined area of the display (the user does not turn his eyes away from the display) 
("No" at step S26) and the situation detection section 1 detects that the non-user infringes 
the service of the user ("Yes" at step Sll). In this case, the visual line detection section 
16 keeps watch for movement of the visual line of the non-user. If the direction of the 
visual line of the non-user points to the predetermined area of the display (the non-user 
watches the display behind the user) ("No" at step S27), the situation detection section 1 
outputs the visual line information to the service supply permission section 2. The 
service supply permission section 2 controls [controlls] the service supply section 3 to 
interrupt the supply of the service (step SI 2). 

[0083] In this way, in the tenth embodiment, detection of the visual line of the user 
and the non-user controls [controlls] the supply of the service. Therefore, the security of 
the service for the user is strengthened. 

[0084] In the eighth, ninth, tenth embodiments, the visual line of the user or the non- 
user is detected. However, by detecting a direction of a face of the user or non-user, the 
supply of the service may be controlled. In this case, as for the user, if the direction of 
the user's face does not point to the display, the supply of the service is interrupted. As 
for the non-user, if the direction of the non-user's face points to the display, the supply of 
the service is interrupted. As for the method , for detecting the direction of the face, the 
face area is extracted from the image including the user or the non-user. The direction of 
the face is pursued according to the face area. 

[0085] In the above embodiments, the image is used for detecting the person 
information. However, sound or wireless may be used for detecting the person 
information. In case of the sound, the user is specified by the user's voice existing in the 
service-use area. If the user's voice is continuously detected, it is decided that the user is 
under a situation to be supplied the service. If a voice other than the user's is detected, it 
is decided that the security of the service of the user is infringed. In case of the wireless, 
the user is specified by detecting ID information of a wireless card. The user previously 
brings the wireless card with him. If the ID information of the user's wireless card is 
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continuously detected, it is decided that the user is under a situation to be supplied the 
service. If ID information other than the user's wireless card is detected, it is decided that 
the security [sequrity] of the service of the user is infringed. 

[0086] FIG. 17 is a schematic diagram showing a method of supplying a software for 
the present invention. A series of processings of the service supply permission section 2 
(shown in FIGS. 3-6, 8-10, 12, 14, 16) and necessary information may be stored in a 
memory device as shown in FIG. 17. In this case, it is possible that the memory device is 
applied for each apparatus or content of the memory device is transmitted to each 
apparatus by a communication 5 device. 

[0087] A memory can be used to store instructions for performing the process 
described above, such a memory can be a CD-ROM, floppy disk, hard disk, magnetic 
tape, semiconductor memory, and so on. 

[0088] Other embodiments of the invention will be apparent to those skilled in the art 
from consideration of the specification and practice of the invention disclosed herein. It 
is intended that the specification and examples be considered as exemplary only, with the 
true scope and spirit of the invention being indicated by the following claims. 
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Claims: 

What is claimed is: 

1. A security apparatus for a device supplying a service to a user in a service 
use area surrounding the user, comprising: 

image input means for continuously inputting an image to monitor the 
service use area; 

person discrimination means for continuously recognizing a person in the 
input image, and for registering the person as a user allowed to use the service if the 
person is recognized as an authorized user; 

use situation decision means for deciding that the user is not under a 
situation to use the service in case the user is not recognized in the input image; 

infringement situation decision means for deciding that a security of the 
service use area is infringed in case at least one person other than the authorized user is 
recognized in the input image; and 

service control means for supplying the service to the authorized user and 
for controlling a supply of the service if said use situation decision means decides that the 
user is not under the situation to use the service or if said infringement situation decision 
means decides that the security of the service use area is infringed. 

2. The security apparatus according to claim 1, wherein said service control 
means finishes the supply of the service in case said use situation decision means decides 
the user is not under the situation to use the service. 

3. The security apparatus according to claim 1, wherein said service control 
means interrupts the supply of the service until the infringement situation is relieved in 
case said infringement situation decision means decides the security of the service is 
infringed. 

4. The security apparatus according to claim 1, wherein said service control 
means decides whether work of the user for the service is completed or not in case said 
use situation decision means decides the user is not under the situation to use the service. 
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5. The security apparatus according to claim 4. wherein said service control 
means finishes the supply of the service in case the work of the user for the service is 
completed, interrupts the supply of the service in case the work of the user for the service 
is not completed, and relieves an interruption of the supply of the service in case said use 
situation decision means decides the user is under the situation to use the service again. 

6. The security apparatus according to claim 1, wherein said person 
discrimination means recognizes the user by referring to a person comparison dictionary 
to recognize persons allowed to use the service. 

7. The security apparatus according to claim 1, wherein said person 
discrimination means generates a person comparison dictionary to recognize unspecified 
users allowed to use the service, and recognizes the unspecified person by referring to the 
person comparison dictionary while the unspecified person is a user. 

8. The security apparatus according to claim 1, wherein said service control 
means sends a warning to the user when said infringement situation decision means 
decides the security of the service is infringed. 

9. The security apparatus according to claim 8, wherein said service control 
means controls the supply of the service in accordance with an indication of the user 
being warned. 

10. The security apparatus according to claim 1, wherein said service control 
means controls the supply of the service in accordance with security degrees preset to a 
unit of the service or information for the service. 

11. The security apparatus according to claim 1, wherein said service control 
means detects movement of the visual line or a direction of the face of the user and 
controls the supply of the service in accordance with the movement of the visual line or 
the direction of the face of the user. 
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12. The security apparatus according to claim 1, wherein said service control 
means detects movement of the visual line or a direction of the face of the non-user and 
controls the supply of the service in accordance with the movement of the visual line or 
the direction of the face of the non-user. 

13. A security apparatus for a device supplying a service to a user, 
comprising: 

person discrimination means for recognizing a user requesting the service; 

use situation decision means for deciding whether the user is under a 
situation to use the service; 

infringement situation decision means for detecting whether a non-user 
intrudes into a use area of the service to decide whether the service is infringed; and 

service control means for supplying the service to the user in case said 
person discrimination means recognizes the user, and for controlling a supply of the 
service if said use situation decision means decides the user is not under the situation to 
use the service or if said infringement situation decision means decides that the security 
of the service is infringed, 

wherein said service control means decides whether work of the user for 
the service is completed or not in case said use situation decision means decides the user 
is not under the situation to use the service, 

wherein said service control means finishes the supply of the service in 
case the work of the user for the service is completed, interrupts the supply of the service 
in case the work of the user for the service is not completed, and relieves an interruption 
of the supply of the service in case said use situation decision means decides the user is 
under the situation to use the service again, and 

wherein said service control means finishes the supply of the service in 
case said use situation decision means decides the user is not under the situation to use 
the service within a predetermined time during the interruption of the supply of the 
service. 

14. A security apparatus for a device supplying a service to a user, 
comprising: 

person discrimination means for recognizing a user requesting the service; 
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use situation decision means for deciding whether the user is under a 
situation to use the service; 

infringement situation decision means for detecting whether a non-user 
intrudes into a use area of the service to decide whether the service is infringed; and 

service control means for supplying the service to the user in case said 
person discrimination means recognizes the user, and for controlling a supply of the 
service if said use situation decision means decides the user is not under the situation to 
use the service or if said infringement situation decision means decides that the security 
of the service is infringed, 

wherein said service control means sends a warning to the user when said 
infringement situation decision means decides the security of the service is infringed. 

wherein said service control means controls the supply of the service in 
accordance with an indication of the user being warned, and 

wherein said service control means controls the supply of the service in 
accordance with predetermined control information in case of a non-response of the user 
within a predetermined time after warning of the infringement. 

15. A security method associated with supplying a service to a user in a 
service use area surrounding the user, comprising the steps of: 



recognized as an authorized user; 

supplying the service to the authorized user; 

deciding that the user is not under a situation to use the service in case the 
user is not recognized in the input image; 

deciding that a security of the service use area is infringed in case at least 
one person other than the authorized user is recognized in the input image; and 

controlling the supply of the service if the user is not under the situation to 
use the service or if the security of the service use area is infringed. 

16. A computer readable memory containing computer-readable instructions 
to supply a service to a user in a service use area surrounding the user, comprising: 

Library: Los Angeles; Document #: 401 65vl -22- 
43701.00034 



continuously inputting an image to monitor the service use area; 

continuously recognizing a person in the input image; 

registering the person as a user allowed to use the service if the person is 



•» # 

instruction means for causing a computer to continuously input an image 
to monitor the service use area; 

instruction means for causing a computer to continuously recognize a 
person in the input image; 

instruction means for causing a computer to register the person as a user 
allowed to use the service if the person is recognized as an authorized user; 

instruction means for causing a computer to supply the service to the 
authorized user; 

instruction means for causing a computer to decide that the user is not 
under a situation to use the service in case the user is not recognized in the input image; 

instruction means for causing a computer to decide that a security of the 
service use area is infringed in case at least one person other than the authorized user is 
recognized in the input image; and 

instruction means for causing a computer to control a supply of the service 
if the user is not under the situation to use the service or if the security of the service use 
area is infringed. 
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SECURITY APPARATUS AND METHOD 
BACKGROUND OF THE INVENTION 

1. Field of the Invention 

The present invention relates to a security apparatus and 
a method for supplying a service to a user by confirming a 
use situation of the user and infringement situation of 
non-user in order to keep safety and reliability for the 
service. 

2. Description of the Related Art 

A method to specify the user in service-use area by 
electrical means and to execute security for each kind of 
service is very important in present electrical society. Vari- 
ous kinds of security means, for example, memorized 
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Further in accordance with the present invention, there is 
provided a security method associated with supplying a 
service to a user, comprising the si:ep of: recognizing a user 
desiring the service: supplying the service to the user if the 
user is recognized; deciding whether the user is under a 
situation to use the service; deciding whether a non-user 
intrudes into a use area of the seivice; and controlling the 
supply of the service if the user is not under the situation to 
use the service or if the non-user is intruding into the use 
area of the service. 

Further in accordance with the Resent invention, there is 
also provided a computer readable memory containing 
computer-readable instructions to supply a service to a user, 
comprising: instruction means for causing a computer to 



ous kinds of security means, for example, memorized l$ f ^ a user desiring the service; instruction means for 
number, magnetic card, IC card, wireless card, sound causing a com putei to supply the service to the user if the 
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comparison, fingerprint comparison, retina comparison and 
face comparison, have been used. However, the memorized 
number, the magnetic card, the IC card and the wireless card 
specifies the user by information only the user knows or with 
a memory device key for only the user. Therefore, they are 
in danger of being used unjustly. 

the other hand, sound comparison, fingerprint 
comparison, retina comparison, and face comparison are 
effective^ersonal confirmation means to defect an other's 25 
unjust use because the user's physical features are used as a 
key. However7u\epersonal confirmation is only used at the 
begining of serYiceN^fter the service begins, the service can 
not be interrupted. TheVfore. even if an other person changes 
for the user after the serwecis begun, the service can not be 30 
interrupted. In short, this kinaVpf service includes danger that 
the other person utilizes the seYyice unjusty. This kind of 
problem is included in the memorial number, the magnetic 
card, the IC card and the wireless 
^^jjTJ Ag ~ m ■ H ihmre. thft known s e curity apparatu ses 35 



causing a computer to supply the service to the user if the 
user is recognized; instruction means for causing a computer 
to decide whether the user is under a situation to use the 
service; instruction means for causing a computer to decide 
20 whether a non-user intrudes into a use area of the service; 
and instruction means for causin g a computer to control a 
supply of the service if the user is decided not under the 
situation to use the service or the non-user is decided to 
intrude into the use area of the service. 
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include following problems 

(1) After security is cancelled by confirming the user, 
security function is not executed while supplying the ser- 
vice. Therefore, even if an other person changes for the user, 
the service is still supplied continuously. 

(2) While supplying the service to the user, it often 
happens that an other person watches information not to be 
disclosed on display from behind the user. This problem 
occurs because security is not provided in the physical space ^ 
surrounding the user and the apparatus even if security is ^; 
executed inside the apparatus electrically. Especially, this 
problem happens when the service is supplied to the user in 
public place in which other persons are not excluded. 

SUMMARY OF THE INVENTION 



BRIEF DESCRIPTION OF THE DRAWINGS 

FIG. 1 is a block diagram of the security apparatus 
according to the present invention. 

FIG. 2 is a block diagram of the situation detection section 
of the security apparatus according to the present invention. 

FIG. 3 is a flow chart of processing of a service supply 
permission section according to a first embodiment of the 
present invention. 

FIG. 4 is a flow chart of processing of the service supply 
permission section according to a second embodiment of the 
present invention, 

FIG. 5 is a flow chart of processing of the service supply 
permission section according to a third embodiment of the 
present invention. 

FIG. 6 is a flow chart of processing of the service supply 
permission section according to u fourth embodiment of the 
present invention, 

TTr-*^lo<£diagram of the sequrity apparatus 
according to a fifth emTJoTtaetft-oiihcjaresent invention. 



FIG. 8 is a flow chart of processing of the service supply 
permission section according to Ihe fifth embodiment of the 
present invention, 
ffi^ / F I G. 91 s-frfloaLchart^prc<:essing of the service supply 
It is an object of the present invention to provide a y> 0 p^^^Q^ sectionaccoro^n^to^ixth embodiment of the 
security apparatus and method to supply a service to the user pressent invention, 
while safety and reliability for the service are kept. j^q jn is a flow chart of processing of the service supply 

According to the present invention, there is provided a permission section according to a seventh embodiment of 
security apparatus for a device supplying a service to a user. 55 the present invention. 

comprising: person discrimination means for recognizing a piG^ Tl \j a blo rJc dim-in m of the sequrity apparatus 
user requesting the service; use situation decision means for ^according to a eighuTembo^rfefl^of the present invention, 
deciding whether the user is under a situation to use the ^ ^ of s - of the servicc ply 

service; infringement situation decision means for detecting section accordi ng t0 ^ cighth cmbodimcnt 0 f 

whether a non-user intrudes into a use area the service to 60 * nrMM : nv . ntinn 



decide whether of the service is infringed; service control 
means for supplying the service to the user in case said 
person discximination means recognizes the user, and for 
controlling a supply of the service if said use situation 
decision means decides the user is not under the situation to 
use the service or if said infringement situation decision 
means decides that the security of the service is infringed. 
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the present invention. 

FIG. 13 is a schematic diagram of detection of visual lines 
of the user in the image according to the eighth embodiment 
of the present invention. 

FIG. 14 is a flow chart of procsssing of the service supply 
permission section according to a ninth embodiment of the 
present invention. 
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FIG. 15 is a schematic diagram of detection of visual lines 
of the non-user in the image according to the ninth embodi- 
ment of the present invention. 

HG. 16 is a flow chart of processing of the service supply 
permission section according to a tenth embodiment of the 
present invention. 

FIG. 17 is a schematic diagram of supply method for ™ c Gjf^ 
security apparatus by software according to the present^^ ^ ^ 

invention. io user< defecting the use situation, or detect an infringement 

situation-Tfoe situation detection ;;ection 1 is comprised of 
an image input section 11. a person area detection section 12. 
.... a person comWson section 13, a use situation decision 
The embodiments of the present invention are described section 14 an d atfinfringement siaiation decision section 15. 
below with reference to the drawings. FIG. 1 is a block l$ ^ ^ jmputlcrtion 11 inputs an image through a TV 
diagram of the security apparatus according to the present camcra The TV cam&ta is set in the service use area for 
invention. The security apparatus of the present invention is 



anoGreroerson, the supply of the service interupts untU the 
iimingemfeB^relieved. The service supply section 3 sup- 
plies various Mnafrots^rvice under the control of the service 
supply permission sectioiT^or sxample. an uninhabited 
ATM (antomatic teller's machine>saportable terminal, or. a 
computer used by several persons ma^>mprise the service 
supply section 3. 

jIG. 2 is a block diagram of the i ituation detection section 
. IiNhis case, an image is used as a way to Discriminate the 



DETAILED DESCRIPTION OF THE 
PREFERRED EMBODIMENTS 



continuously observing^ user to be supplied the service. 
The person area detection section 12 analyzes images input 
by the image input section llsand monitors the secure area 
20 from the input image. For exarnpkj, a difference between the 
input image and background imaf^uninhabited area) indi- 
cates a person present in the seoireSarea. The person area 
detection section 12 outputs pen;on detection information 
Dl when the person present is cLetectedSand outputs non- 
infringement situation is outputted to the service supply 2$ Marmatioil D i w hen the persc\present is not 
permission section 2 as person information. In this place, the Steeled. ^ 
situation that the user can utilize the service represents that 



comprised of a situation detection section 1, a service supply 
permission section 2 and a service supply section 3. The 
situation detection section 1 discriminates the user in 
service-use area, detects whether the user is under a situation 
to use the service (use situation) and detects whether a 
non-user infringes the security of the user (infringement 
situation). The detection result of the use situation and the 
infringement situation is outputted to the service supply 
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the user is confirmed to be permitted the supply of the 
service and the user continuously utilizes the service. A 
situation that the user continuously utilizes the service 
represents a status that the user actually interacts with the 
service application (the user inputs operation to system or 
the user is supplied output from the system). A status of 
non-interaction within predetermined time indicates 
thinking, chattering, looking away. Another status indicates 
when the user moves away from service use-space within a 
predetermined time. 

The situation that a non-user infringes the security of the 
user represents that a person other than the confirmed user 
intentionally or accidentally uses the service while the tf C 
service is supplied to the confirmed user. The other person 
can then interact with the operation or information to be 
disclosed only to the confirmed user. When the user interacts 
with the service application of the system through a display, 
the other person looks on the display behind the user. 
Otherwise, the other person intends to interact with the 
service application in the user's absence within the prede- 
termined time. 
In order to detect each situation, the situation detection 
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detected. 

The person comparison section 13 compares the person 
present area image with the content of a person comparison 
dictionary to specify which person is present in the secure 
area image. The person comparison dictionary previously 
stores at least comparison information of persons permitted 
to use the service. The person comparison section 13 outputs 
person discrimination information D2 representing whether 
the person present coincides with a person permitted to use 
the service. If the person present coincides with a person 
permitted to use the service in the person comparison 
dictionary, the person present is registered as the user who 
can be supplied the service. 

ise situation decision section 14 detects whether the 
user is unfekasituation capable of using the service. After 
the user of thep&sea^arca is registered and if the registered 
user is now included m^thejjerson discrimination informa- 
tion D2. the user is decideci^teMx capable of using the 
service. The use situation decison section 14 outputs use 
situation information D3 representing wifethcr the user is 
capable of using the service. 

The infringement situation decision section 15 detects 
whether a non-user infringes the security of the service. If a 



section 1 monitors service-use area using image .or sound. 50 PH*» *an the user uUizing the service now is 

_ .... .... • _ ■ ♦ inMnrloH in rfc* nPrcnn rfi «trn minatinn information D2. the 
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included in the person discrimination information D2, the 
infringement situation decision section 15 decides that the 
security of the service is infringed and outputs infringement 
situation information D4 indicates a breach of security. The 
55 person detection/non-detection information Dl, the person 
discrimination information D2, the use situation information 
D3. the infringement situation information D4 are called 
person information. The person information is read out by 
;c tion section i. in mis case, uic me service supply pennission sc:ction 2 to TO. 3 

section 2 starts the supply of the 60 explained afterward) and referred at steps S3, S4. S*. Sll in 
in the service-use area. *K3* 3 * 



Especially, the use situation of the user and the infringement 
situation of the security are detected by the image (person 
detection/direction of face detection/person confirmation) 
and the sound (person detection/person specification/ 
chattering detection). Monitoring may be wireless. 

service supply permission section 2 controls start/ 
"Cs* mtern5f)Uon/inteiTuption and cancellation/end of the service 
for the service supply section 3 according to the detection 
result of the situation detection section 1. In this case, the 
service supply 
service when the user 



The service supply permission section 2 controls the supply HG. 3 is a flow chart of processing of the service supply 

of the service in at least one ofme cases where the user does permission section 2. As mentioned above, the situation 

not utilize the service or where ^security of the user is detection section 1 discriminates the user ui the service-use 

infringed Especially when the userNloes not utilize the 65 area and monitors the use situation and the infringement 

service during the supply of the service\tiie supply of the situation using the image inputted from the TV camera. The 

service finishes. If the security of the useNj infringed by person information as detection result of the situation detec- 
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tion section 1 is outputted to the service supply permission 
section 2. By using the person information, the service 
supply permission section 2 executes a security controlling 
process to the service supply section 3. First, the service 

" - . • i-- ^ ^u+ 4 «*«i- fkd n^renn information 



Next, additional embodiments of the present invention 
will be explained. 

(a) ease of monopoly of terminal/improvement of com- 
mon use of terminal (second embodiment) 



supply permission section 2 obtains the person infermatton 5£ /7 ^ anothiff person can not use the 

from the situation detection section 1 (step SI). When the> O tenninal wh jI e TKrrrs^^ the service through the 
supply of the service begins ("No M at step S2), the service tcrmina i. Therefore, the otBer^on must wait to use the 
supply permission section 2 monitors the security area in the same terminal untill the first userTtflishes. In short, in the 
service use area according to the person detection/non- &st embodiment, use by a specified ufcerjor the terminal 
detection information Dl (step S3). If a person area is not 10 excludes use by an other person. 

detected ( 4 W at step S3), processing of the service supply ^0 ^fij^ second embodiment, in case the situation for the user 
permission section 2 is returned to step SI. On the other C^ ot capible^ of using the service is detected before the supply 
hand, if a person is detected ("Yes" at step S3), the service of ^ scr y^fo r user finishes, the supply of the service 
supply premission section 2 decides whether the person is . fi reserved (inSfcupted) and the new user can .be supplied 
permitted as a user (step S4). As a result, in case the person is ^ s&vi<x froJ n^ begining. As a result, during the 
f. > ***** oc a .,««• r-Mn" at sten the service t . * 
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is not permitted as a user ( W at step S4), the service 
supply permission section 2 rejects the supply of the service 
(step SB) and returns processing to step SI. In case the 

person is permitted as a user ("Yes" at step S4), the service agajji j,^^ while 
supply permission section 2 registers the person as the user 20 respectivclVt plunu uscrs 
(step S6). After that, the use situation of the person is ^ $ame tenninal onc 
continuously monitored. The service supply permission sec- 
tion 2 begins the supply of the service (step S7) and returns 
the processing to SI. 

^Oth^hcye process steps (step S1-S7) are the same as 

^ normal conm^rnation of security (from confirming the user to 
begining the suppiy^a^the service). In this place, after the 
supply of the service isoegun. the registered- user exists in 
the service-use area and the^fCEistered _ 
detected at step SI. After the supprW the service is begun, 
the processing is returned to step S8 through steps SI and 
S2. 



interruption of the servfce. a different service is supplied to 
another user. When the 8rMnal user comes back to the 
terminal. the reserved semcei&^iipplied to the original user 
Therefore, while each Tei^ice is not completed 
can be suppbQd each service from 
the* same terminal one after another. AtioUtionally, if the 
original user does not come back: to the sainVterrninal, the 
reserved service is finished and memory area corresponding 
to the reserved service is relieved to save memory resources. 

FIG. 4 is a flow chart of proce ssing of the service supply 
permission section according to the second embodiment. In 
this case, assume that a computer nerworkis commonly used 

^- by plural users and a plurality of terminals connected to the 

person must be netW0 rk provide a security funoion. In FIG. 4, steps S15. 
ic H M .m w S16 S17 S18 m newly added iQ comparison with FIG. 3. 

Each user is confirmed respective ly and supplied the service 
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rj, through a respective terminal (step SI, S2) 
The service supply permission section 2 decides whether ^ ^J^n this place, assume that a user A creates a document 

. n^rcnri U iicino th^ cf>rvirp. arrordin^totheUSe 
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the registered person is using the service according to the use 
situation information D3 (step S8). As a result, if the 
situation of using the service is detected, for example, a 
user's absence continues above a predetermined time ("No* 
at step S8), the service supply permission section 2 termi- 
nates the supply of the service immediately (step S9). Next, 
the registration of the user is cancelled (step SI0) and the 
processing returns to step SI in order to register a new user. 

On the other hand, the service supply permission section 
2 decides whether a non-user infringes the security of the 
user according to the infringement situation information D4 45 tion notice sent by 
(step Sll). As a result, if the situation that a non-user theteminalAandwor: 



jsinc a terminal A and leaves from the tenninal A without 
competing the work. The terminal A detects that the user A 
is not Oring the service ("No" at step S8) and that the supply 
of the service is not completed ("No" at step S15). The 
service of creating the document at the present time is stored 
40 (reserved) in\a memory area of the terminal and the supply 
of the teminalis interrupted (step S16, S17). In this case, the 
situation for tne user not using the service is detected 
according to the use situation information D3. Completion 
situation of the supWy of the service is decided by comple- 
user. Next, a user B is confirmed by 
through the terminal A. Assume that 



5' 



infringes the security is detected, for example, an other when the user A comesNback to the terminal A, the user B 

person watches the display behind the user during the supply goes away from the terminal A The terminal A newly 

of the service ("Yes" at step Sll), the service supply reserves the results of theWvice for the user B (steps S8. 

permission section 2 temporarily interrupts the service to 50 S15.S16.S17)andbegmstHesupply of the reserved service 

maintain security untill the security is not infringed (step to the user A (steps S8, Sll. SW, S14). The memory area to 

512) If a situation that the security is not infringed is store the result of the servicers commonly used by each 
detected during the interruption of the service ("Yes" at step terminal on the network Therefore, if the user B is con- 

513) . the service supply permission section 2 cancels the firmed by a terminal B (steps S4. S^, S7), the reserved result 
interruption of the service and begins the supply of the 55 0 f the service for the user Bis supplied through the terminal 
service (step S14) B (steps S8, Sll, S13. S14). On theVher hand, if the user 



^- .u- «*««iH^ th* cPtvir? i<: A doe's not come back to the teiminalV during a predeter- 

^-j ^a^Jhis way, in case the user permitted tne service is ^ w A 

^peeffieo^the service use area, the supply of the service is 



mined time ("Yes* at step S18). the tentorial A finishes the 
confirmed and supplied money through^ Automatic Teller the service. 

Machine, or the user operates a handy tehmnal by watching 6%Ols mSBanad^bove. « the second embodiment. Je 
secret information on a display, the securirKof the user is su&y* the service^jchuscr.s reserved untill the work 
effectively protected according to the presentkyention. of each user is corapletedTTtJ^short. a ngnt service-use 



02/28/02 16:01 FAX 213 486 9807 



SQUIRE SANDERS LA 



L^] 006 



6,049,875 



8 



„ £ 

unspeciftetf-t^^ sitf- 



w 

fcfl 



! - E 



generatfckbv confirming the user is assured unnll the user s 
work is compteted^Thc user B does not have to wait untill 
the user A fidshes^Tiig^e terminal. By changing the user 
A for the user B, the userXcan use the terminal without 
infhn|gjaeiirftn^he security aNhe user A. 
xtention r< — 4 
in* uicfirst embodiment, in order to be supplied the 
service^theTjseT™ be confirmed as a registered person. 
In short if the user is not previously registered, he can not 
be supplied the service. However, there is a service to be 
used by many and unspecified persons. Therefore, in a thud 
embodiment, the service is supplied to an unspecified person 
without confirmation. At that time, a comparison dictionary 
of the unspecified person is generated and registered. During 
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oftheunspecmedpersonisgeneraieoaiiaic g «^^^ f step S20). After sending the 
the supply of the service, the user (the ^P^) SS^SL. an indica 



fifth embodiment. In FIG. 7. an interactive control section 4 
is a new section in comparison with FIG. 1. FIG. 8 is a flow 
chart of processing of the service supply permission section 
2 according to the fifth embodiment In FIG. 8, steps S20. 
S21 are new added steps in compirison with FIG^S. 

interactive control section 4 receives anifrttca&bo of 
r or the warning of the infri ugement an|Lcontrolls Ihe 
supply ofNhe service according to the indicaSfc>Ihe_£kd 
of the indication is continuatioii/interruption/end of the 
service. The usV selects the kind of the indication by. for 
example, using atey board. In FIG. 8. during the supply of 
the service if the%ttuation detection section 1 detects 
infringement by a nonW ("Yes" at step SU). the service 
supply permission sectionSlsends the warning to the user 
r ■ - ■ ^ rning, the interactive control 



is oiscriminated from other users by referring to the com- 
parison dictionary. Therefore, the service is supplied to 
many and unspecified persons while the security is main- 
tained. 

FIG. 5 is a flow chart of processing of the service supply 
permission section 2 according to the third embodiment In 
FIG 5. step S19 is newly added instead of steps S4, S5 in 
comparison with FIG. 3. When the supply of the service is 
begun ("No" at step S2), the service supply permission 
section 2 detects the person in the service use area according 
to the person detection/non-detection information Dl (step 
S3). If the person is detected ("Yes" at step S3), the service 
supply permission section 2 generates the person compari- 
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section 4 receives an indication><the user (step S21), the 
service supply permission section ^controlls the supply of 
the service according to the indication 

If the user indicates continuation of the service 
("CONTOTUAITON" at step S2;i), the interactive control 
section 4 outputs the indication of continuation to the service 
supply permission section 2. The service supply section 3 
continues to supply the service according to the indication of 
continuation. 

If the user indicates interruption of the service 
("INTERRUPTION" at step.S2:L), the interactive control 
section 4 outputs the indication oi interruption to the service 
supply permission section 2. Tht; service supply section 3 
dictionary (step S19) and registers the person as a new „ interrupts the supply of the service according to the indica- 
user (step S6). After that, the use situation of the new user 30 tio n of ^ejntgru ption. 

^continuously monitored, and the service supply permis- ^Sfff^^m^A^^ ^SntaS 
sion section 2 begins to supply the service (step S7). In this ^ S21). tfc interract^ 

way theserviceissuppUedtoanunspecifiedpersonwithout of endV^-**^ supply penKs^ion^cUon 2. The 
confirmation and the person comparison dictionary is gen- service supply section 3 finishes the suppl^ef the service 

- — - way when the uscr receives the warning of 

ng&aent, the user can select continuaUon/interruption/ 
end of servicVijv using interactive function. As a result if 
the user works through equipment in an office, whenever an 
office colleague (pers&Knot necessary for security) stands 
behind the user, interruption^* the work is excluded. In 
short, a change for the worse ofbse-feeling of the equipment 
is avoided. Otherwise, when using ^he equipment in a public 
place, interruption of the service is indicated by the user for 
the warning of a non-user standing beriinXthe user. 

In the fifth embodiment, the user can select the indication 
to keep security in case of infringement of the security. After 
selecting the indication, the apparatus (equipment) can auto- 
matically keep the security of the user. In this case, if the 
warning of the infringement is outputted as an image, the 
user understands circumstances firound the user by watching 
the image (the user and non-user are included). The user can 
select the indication (continuation/interruption/end of the 
service) according to the circumstances. 

(e) Default action of neglecting the warning (sixth 
embodiment) 

In the sixth embodiment, in case the indication is not 
selected by the user within predetermined time after gener- 
ating the warning, the apparatus automatically executes 
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erated to be registered. Therefore, a new user except far 
pre-registered user is able to be supplied the service. In this v 
case, during the supply of the service for the new user, he is 
discriminated from another person by referring the person 
comparison dictionary in order to maintain the security, 
(c) Warning of infringement (fourth embodiment) 
In the fourth embodiment, when an infringement situation 
of the security is detected, a warning is sent to the ujefTIn 
this casar=tbe service supply permission section 2^ontrolls 

ffcgining\nd end of the service and does not ex 
intefeiptiojujBd cancellation of the service in case of 
infringement of the security. 

FIG. 6 is a flow chart of processing of the service supply 
permission section 2 according to the fourth embodiment. In 
FIG. 6, step S20 is a newly added step instead of step S12 50 
in comparison with FIG. 3. First, when the situation detec- 
tion section 1 detects the infringement situation of non-user 
("Yes" at step Sll). the service supply permission section 2 
sends a warning of the infringement to the user (step S20). 
Concretely speaking, the warning is outputted through an 55 
image (warning sentence/mark, controlling of color/ 
brightness/blink) or sound (speaking of warning sentence/ 
warning buzzer). When the user receives the warning, it is 



possible for the user to protect his security by, for example, — rr—. r r — : -— - - * 

tu^coffapowersupplyoftheterrr^^ 60 predetermined (default) controlling of the supply of the 

of the terminal. service. This predetermined controlling is freely set by the 

(d) Interactive controlling in case of warning (fifth user. 

embodiment) J* V? FIGJ ^5i floW P roc£SSU1 8 of the service supply 

In the fifth embodiment, by using input means (key board) C^m^^M FIG. 9, ^ps S 20.S21522 S S23^S24 
and output means (display), the supply of the service is 65 *'^ slc P* in ^^ 

SntrollTd by the user's oration for the warning. FIG. 7 is of the security ar^atul^e f iflh embodiment is the same 
a block diagram of the security apparatus according to the as in FIG. 7. In FIG. 9, diirin^he supply of the service, if 
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situation detection section 1 detects an infringement 
ia tion ("Yes" at step Sll). the service supply penmssion 
1 2 sends the warning to the user (step S20). After the 
ig the interactive control section 4 receives indication 
of theXuser. In this case, if no indication °f ^ "mtis 
receiveX above predetermined time (step S22. S23). the yP- 
interact^ control section 4 outputs J^urport of no £1^ 
indication 10 the service supply pemisWrsection 2- The 
service suprifc permission section pontrdls^fee supply of 
the service according to a predeterrWfc2SS6&ol (step S24). 
For example, it continuation of the service is set as the 
predetermined IWol ("CONTINUATION" at step S24). 
the service supplyVction 3 continues to supply the service. 

If interruption of tfle service is set as the predetermined comMJUa uic =>u FF v ™ «. 
control ('INTERRUPTION" at step S24). the service supply 15 dcgree sct ^ ^ ^ of 

section 3 interrupts the\supply of the service. If end ofthe - . - — - ^ n 

service is set as the predetermined controlling ("END" at 
step S24). the service supply section 3 finishes the supply of 
the service. On the other hand, if the indication of the user 
is received within the pred^ermined time ("Yes" at step 
S22). the interactive control section 4 receives the indication 
(step S21) and the service suWy permission section 2 
controlls the supply of the servicV according to the indica- 
tion. In this case, if the user indicates continuation of the ^ ^ 

service ("CONnNUAITONr at ste\ S21). the interactive 25 * T^j* ^""the "mail is able to remain disclosing, 
control section 4 outputs the continualipn indication to * e C V ^7^w uscr indicates a strict secret at the headc 
service supply permission section 2. Tfoe service supply ^ ^ ^T*m„,:^™,*^;*,^™ rti* Hr>rn merit rr 

section 3 continues to supply the servic e\lf the user indi- 
cates interruption ofthe service ("INTERRUPTION" at step 
S21). the interactive control section 4 outoulsthe interrup- 
tion indication to the service supply r^rrrussiiwi section 2. 
Tne service supply section 3 interrupts the supply of jhe 
service. If the user indicates end of the service (\ENET at 
step S21). the interactive control section 4 outputs\foe end 
indication to the service supply permission section X. The 
service supply section 3 finishes the supply of the serVice. 

In this way, if the user neglects a warning of infringement 
the security of the user is kept according to a predetermined 
control (action of default). Otherwise, it is possible to 
control the supply of the service different form the prede- 
termined control for some reasons by a user's indication in 
case of the warning. For example, in an office, as for the 
warning generated whenever a colleague stands behind the 
user operating the equipment, the user does not have to 



document. While the user creates the document whose 
security degree is low, the security degree of the document 
grows by adding another important document In this way. 
the security degree is dynamically changed according to the 
kind of the service/iriformation. 

the seventh embodiment, the security degree is 
changed by unit of the service/infc rmation. FIG. 10 is a flow 
chart ofwocessing of the service supply permission section 
2 accOTdinVio the seventh embodiment In FIG. 10. step S25 
added betwe\s n md S12 is a diffcrent ste P 111 comparison 
with FIG. 3. During the supply of the service, if the situation 
detection section^detects infringement of the security 
("Yes" at step Sll). ffibsservice supply permission section 2 
controlls the supply of th^ervict: according to the security 

^ V;c/information (step S25). In 

this case, it may be possible toaKte security degree is set to 
a mail tool to display the mail burbot set to a WP tool For 
example, when another person watch>sjthe display behind 
the user, the mail tool is only intirrrupted^the mail window 
20 is closed) and the WP window is continuously opened. 
As for setting the security degree, for example, the user 
sets disclosure-possible at the head portion of mail to be 
sent. A window showing the mail tool does not react to the 
infringement of the security. As for both sending side or 



header of a 

documenfclfaeWP window to display the document reacts to 
the infringemehK^f security. For example, the sequrity 
degree for creatmgm>dpcument is dynamically changed by 
adding other imr>ortant d&smieiit. In this way, even if the 
service is supplied to the samesjbject. the supply of the 
service is interrupted or continued f^bQording to the content 
(for example, regular document or strict^ccret document), 
(g) detection of visual line 1 {eighth embodiment) 
In the above embodiments, when another person watches 
the display behind the user creating a document, the 
infringement of security of the user is detected However, it 
is not necessary to pay attention to the other person, but to 
40 pay attention to the visual line (or line of sight) of the user. 
For example, when the visual :ine of the user leaves the 
screen of the display, the window may be closed. In the 
eighth embodiment by detecting a visual line of the user as 
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use situation, the supply of the service is interrupted when 

user operating uic cuiuLuu^im mv wow. — ~- * — t ** 

indicate continuation of the service. Ttie predetermined 45 the visualise leaves the screen. ■ 

control is already set as continuation of the service. In this FIG. 11 is a block diagram of the situation detection 

case the user can continue the operation while neglecting section 1 according to the eighth embodiment. A visual line 

the warning In case the user operates the equipment in a detection section 16 is an added section in comparison with 

public place, the predetermined control is set as interruption FIG- 2. The visual line detection section 16 detects move- 

of the service. In this case, security of the user for the 50 ment of the visual line of the user in case the user 

, , ., . , . . „.___.:_ — / j\ '~ <~nln/)«^ I n th» r5>rcnn riisrr 1 mi nation infor- 

equipment is protected while he neglects the warning. 

(f) Security degree by unit of service/information (seventh 
embodiment) 

In the above embodiments, predetermined security is 
activated for service or information. However, security 
degree is actually different by unit of service/information. If 
a fixed security degree is activated, useless interruption of 
the service is often executed. This happens in case of plural 
workers using information equipment. Accordingly, it is 
desired that the security is activated according to the kind of 60 
the service/information. For example, assume that electric 
mail is transmitted using a PC (personal computer) or a 
document is created using WP (word processor). Plural 

services are then supplied on the display of the equipment. * a^v***m 6 — . . , * 

The electric mail represents a private letter and its security 65 is added instead of step Sll jn compan son with FIG 3. 
degree is high. On the other hand, the security degree of the During the supply of the service, assume that the situauon 
document by WP differs according to the kind of the detection section 1 detects that the user is under a situation 
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(registered) is included in the jcrson disaimination infor- 
mation D2. For example, the face area of the user is 
extracted from the image and the eye area is extracted from 
the face area. The visual line is pursued according to the eye 
area, In this case, if a direction of the visual line is toward 
a predetermined area (visual ari;a of the screen), an indica- 
tion is outputted. If the direction of the visual line is not 
toward the predetermined area,. mother indication is output- 
ted. These indications represent visual line information D5. 

In this way, in the eighth embodiment, the visual line 
information D5 is outputtedto the service supply permission 
section 2 as one of the person information. FIG. 12 is a flow 
chat of processing of the service supply permission section 
2 according to the eighth embodiment. In FIG. 12. step S26 
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to be supplied the service ("Yes" at step S8). Id this tase. the 
visual line detection section 16 keeps watch for the move- 
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visual line detection section 10 Keeps waiui i^ «^ -v. 
ment of the visual line of the user. If the direction of the ^ 
visual line departs from the predetermined area of the 
display ("Yes" at step S26). the situation detection section 1 
outputs the visual line information D5 to the service supply 
permission section 2. The service supply permission section 
2 controls the service supply section 3 to interrupt the supply 
of the service (step S12). HG. 13 is a schematic diagram of 
detection of the visual lines of the user in the image 
according to the eighth embodiment As shown in FIG. 13. 
the visual lines of the user are detected from position of 
black eyes of the user's face in the image. 
j6 > «7 STtmrwaywnthe eighth embodiment, detection of the 
y Qf P visual line ofthTKeT^e^^ the supply of the service. 
Therefore, when another pa^rm^walks behind the user, 
security of the survice is kept onlv>the visual line of the 
user departs from the display 



■ - (i) Detection of visual line 3 (tenth embodiment) 
■5^7 the tenth embodiment, detection of the visual line of the 
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(h) Detection of visual line 2 (ninth embodiment) ^ 



user>nd the non-user controlls the supply of the service. 
FIG. InYis a flow chart of processing of the service supply 
perraissic\section 2 according to the tenth embodiment. In 
FIG. 16, steX S26 and step S27 arc added steps in compari- 
son with FI(A3. During the supply of the service, assume 
that the simatio\detection section 1 detects that the user is 
under a situation^ be supplied tie service ("Yes" at step 
S8) In this case, thcNwsual line detection section 16 detects 
movement of the visuSuine of the user. If the visual line of 
the user departs from theNpredeta mined area, for example, 
the user turns his eyes awajsfrom the display ("Yes" at step 
S26), the situation detection section 1 outputs the visual line 
information D5 to the service sWly permission section 2. 
Then, the service supply permissifcui section 2 controlls the 
service supply section 3 to interrupt the supply of the service 
(ste P> S12), 

_ fetttt other hand, assume tha: the direction of the user 
pofnts to luWedetermined area oj: the display (the user does 
not turn hise^es away from the display) ("No" at step S26) 
and the simationMetection section 1 detects that the non-user 
infringes the servicW the user ("Yes" at step Sll). In this 
case, the visual line detection section 16 keeps watch for 
movement of the visual line of trm non-user. If the direction 
of the visual line of the noiNuser p oints to the predetermined 
area of the display (the non-uier watches the display behind 
the user) (*W at step S27). the\situation detection section 

1 outputs the visual line information to the service supply 
permission section 2. The service sur^lv permission section 

2 controlls the service supply sectionNj to interrupt the 
supply of the service (step S12). 

in the tenth embodiment detection of the 
visual line of file^rs^ndthe noa-user controlls the supply 
of the service. Theref breTth^c urity of the service for the 
user is strengthened. 

In the eighth, ninth, tenth embodiments, the visual line of 
the user or the non-user is detected However, by detecting 



igbth embodiment, a direction of the visuailine of 
the useTcBntrolls the supply of the service. However, if 
another personraoks furtively at the display behind the user, 
the supply of the settee can not be controlled. If the user 
turns his eyes away frbXthe display whenever the other ^ 
person walks behind the ush^the supply of the service is 
interrupted to every time.^erefore. in the ninth 
embodiment, detection of the visuakjine of the non-user 
controlls the supply of the service. 

FIG. 14 is a flow chart of processing of the service supply 30 
permission section 2 according to the ninth embodiment In 
FIG. 14. step S27 is added step between step Sll and step ^ 
S12 in comparison with FIG. 3. In this case, the visual line ^ 
detection section 16 detects the visual line of the non-user Jg^/. ^ 
In case the person is not discriminated by referring to the 35 
person cascrimination information D2 (the person is decided 
to be a non-user), the movement of the visual line of the 
non-user is detected. In the same way as in the eighth 

embodiment, the face area is extracted ^ ato'ectionof aface of theuseror non-user. the supply of the 

indudiog the non-us^ 40 

face area. The visual line of ^ ^ ™ « ^ * \ SSKteJK face does not point to the display, the 
direction of d* visual hoc of scrvice is interrupi:ed As for the non-user, if 

predetermined area (visible area rf " ^ the Section of the non-user's face points to the display, the 

tion is outputted If the direction of the ^ J^e rflte S^f the service is interrupted As for the method for 
non-user does not point to the ^^^^fZ 45 deKuTeS 

indication ^T^^ W **»dHg the user or the non-user. The 

visual line information D5 During th : supply -of JJc service rr^ * s according to the face area, 

assume that the situation detection section 1 detects that the « oirecuon oi mc »«= i v c A ^^ na 

non-user infringes the security of the non-user ("Yes" at step<fo>7 £te above embodiments, the image is used for ^toecting 
111). In this case, the visual line detection section 16 keeps 5 o the pe^iformaUon. However sound or wireless may be 
watch for the move of the visual line of the user. If the used for Wring the person itformation. In case of the 
Section of the visual line of the non-user points to the sound , the is specified by the user's voice existing n 
JSS^LdLef for example, the non-usef watches the the J the us=r's voice is 

display behind the user (W at step S27). the situation detected, it is decWat the user is _ u ^- tt ^^ l J* 
detection section 1 outputs the visuailine information to the 5 5 supplied the service^* a voice other than the user s is 
SSke supply permission section 2. The service supply detected, it is deddedW the security of the service of the 
^IsioZ^l controlls the service supply section 3 to user is infringed. b M W *^£"™gf^ 
Serrupt the supply of the service. FIG. 15 is a schematic by detecting ID informaUoV^ * • ™< us * 

diagramofdetectionofthevisuallkesofthenon-userin^^ previously brings the wtfde>scard wtih tain. H the ^ID 
image according to the ninth embodiment As showninFIG. « infonnation of the user's wir^ss card is continuously 

° ~ j j £. . J-t~~+*A It A.»r*\AaA that fhp n«: 



jjuu^v p ■ 

15, the visual lines of the non-user are detected from 
position of black eyes of the non-user's face in the image. 

In this way. in the ninth embodiment, the detection of the 
visual line of the non-user controlls the supply of the service. 
Therefore, in case the non-user looks furtively at the display 65 
behind the user, the security of the service for the user is 
kept. 



detected, it is decided that the user\under a situation to be 
supplied the service. If ID inf onnatio\other than the user's 
wireless card is detected, it is decided that the sequrity of the 
service of the user is infringed. 

FIG. 17 is a schematic diagram showing a method of 
supplying a software for the present invention. A series of 
processings of the service supply permission section 2 



